Processing information using a hive of computing engines including request handlers and process handlers

ABSTRACT

A hive of computing engines, typically including clients, request handlers, and process handlers, is used to process of a processing job in order to determine a processing result. A client assigns a processing job to a request handler, which assigns the processing job to one or more processing handlers. One implementation uses one or more volunteer operations to identify the request handler, and/or the one or more process handlers. One implementation updates the request handler with state information related to the partial processing of the processing job such that the completion of the processing job can be assigned to other one or more process handlers, such as, but not limited to, in response to a detected error condition.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of copending nonprovisional application Ser. No.10/236,784, filed Sep. 7, 2002, with this application being herebyincorporated by reference in its entirety.

FIELD OF THE INVENTION

This invention especially relates to processing of informationincluding, but not limited to transactional processing using multiplenetworked computing systems; and more particularly, the inventionrelates to processing information using a hive of computing engines,typically including request handlers and process handlers.

BACKGROUND OF THE INVENTION

Many businesses are demanding faster, less expensive, and more reliablecomputing platforms. Brokerage houses,, credit card processors,telecommunications firms, as well as banks are a few examples oforganizations that require tremendous computing power to handle acountless number of small independent transactions. Currently,organizations that require these systems operate and maintainsubstantial servers. Further, the cost associated with these machinesstems not only from the significant initial capital investment, but thecontinuing expense of a sizeable labor force dedicated to maintenance.

When it comes to mission-critical computing, businesses and otherorganizations face increasing pressure to do more with less. On onehand, they must manage larger transaction volumes, larger userpopulations, and larger data sets. They must do all of this in anenvironment that demands a renewed appreciation for the importance ofreliability, fault tolerance, and disaster recovery. On the other hand,they must satisfy these growing requirements in a world of constrainedresources. It is no longer an option to just throw large amounts ofexpensive hardware, and armies of expensive people, at problems. Thechallenge businesses face is that, when it comes to platforms formission-critical computing, the world is fragmented. Different platformsare designed to satisfy different sets of requirements. As a result,businesses must choose between, and trade off, equally importantfactors.

Currently, when it comes to developing, deploying, and executingmission-critical applications, businesses and other organizations canchoose between five alternative platforms. These are mainframes,high-availability computers, UNIX-based servers, distributedsupercomputers, and PC's. Each of these approaches has strengths andweaknesses, advantages and disadvantages.

The first, and oldest, solution to the problem of mission-criticalcomputing was the mainframe. Mainframes dominated the early days ofcomputing because they delivered both availability and predictability.Mainframes deliver availability because they are located in extremelycontrolled physical environments and are supported by large cadres ofdedicated, highly-trained people. This helps to ensure they do not fallvictim to certain types of problems. However, because they are typicallysingle-box machines, mainframes remain vulnerable to single-pointfailures. Mainframes deliver predictability because it is possible tomonitor the execution and completion of processes and transactions andrestart any that fail. However, the limitation of mainframes is that allmonitoring code must be understood, written, and/or maintained by theapplication developer. The problem mainframes run into is that suchsystems fall short when it comes to three factors of high importance tobusinesses. First, mainframes tend not to offer high degrees ofscalability. The only way to significantly increase the capability ofsuch a system is to buy a new one. Second, because of their demandingnature, mainframes rely on armies of highly-trained support personneland custom hardware. As a result, mainframes typically are neitheraffordable nor maintainable.

Developed to address the limitations and vulnerabilities of mainframes,high-availability computers are able to offer levels of availability andpredictability that are equivalent to, and often superior to,mainframes. High-availability computers deliver availability becausethey use hardware or software-based approaches to ensure high levels ofsurvivability. However, this availability is only relative because suchsystems are typically made up of a limited number of components.High-availability computers also deliver predictability because theyoffer transaction processing and monitoring capabilities. However, aswith mainframes, that monitoring code must be understood, written,and/or maintained by the application developer. The problem withhigh-availability computers is that have many of the same shortcomingsas mainframes. That means that they fall short when it comes todelivering scalability, affordability, and maintainability. First, theyare largely designed to function as single-box systems and thus offeronly limited levels of scalability. Second, because they are built usingcustom components, high-availability computers tend not to be eitheraffordable or maintainable.

UNIX-based servers are scalable, available, and predictable but areexpensive both to acquire and to maintain. Distributed supercomputers,while delivering significant degrees of scalability and affordability,fall short when it comes to availability. PC's are both affordable andmaintainable, but do not meet the needs of businesses and otherorganizations when it comes to scalability, availability, andpredictability. The 1990s saw the rise of the UNIX-based server as analternative to mainframes and high-availability computers. These systemshave grown in popularity because, in addition to delivering availabilityand predictability, they also deliver significant levels of scalability.UNIX-based servers deliver degrees of scalability because it is possibleto add new machines to a cluster and receive increases in processingpower. They also deliver availability because they are typicallyimplemented as clusters and thus can survive the failure of anyindividual node. Finally, UNIX-based servers deliver some degree ofpredictability. However, developing this functionality can requiresignificant amounts of custom development work.

One problem that UNIX-based servers run into, and the thing that haslimited their adoption, is that this functionality comes at a steepprice. Because they must be developed and maintained by people withhighly specialized skills, they fall short when it comes toaffordability and maintainability. For one thing, while it istheoretically possible to build a UNIX-based server using inexpensivemachines, most are still implemented using small numbers of veryexpensive boxes. This makes upgrading a UNIX-based server an expensiveand time-consuming process that must be performed by highly-skilled (andscarce) experts. Another limitation of UNIX-based servers is thatdeveloping applications for them typically requires a significant amountof effort. This requires application developers to be experts in boththe UNIX environment and the domain at hand. Needless to say, suchpeople can be hard to find and are typically quite expensive. Finally,setting up, expanding, and maintaining a UNIX-based server requires asignificant amount of effort on the part of a person intimately familiarwith the workings of the operating system. This reflects the fact thatmost were developed in the world of academia (where graduate studentsare plentiful). However, this can create significant issues fororganizations that do not have such plentiful supplies of cheap,highly-skilled labor.

A recent development in the world of mission-critical computing is thedistributed supercomputer (also known as a Network of Workstations or“NOW”). A distributed supercomputer is a computer that works by breakinglarge problems up into a set of smaller ones that can be spread acrossmany small computers, solved independently, and then brought backtogether. Distributed supercomputers were created by academic andresearch institutions to harness the power of idle PC and othercomputing resources. This model was then adapted to the business world,with the goal being to make use of underused desktop computingresources. The most famous distributed supercomputing application wascreated by the Seti@Home project. Distributed supercomputers have grownin popularity because they offer both scalability and affordability.Distributed supercomputers deliver some degree of scalability becauseadding an additional resource to the pool usually yields a linearincrease in processing power. However that scalability is limited by thefact that communication with each node takes place over the commonorganizational network and can become bogged down. Distributedsupercomputers are also relatively more affordable than otheralternatives because they take advantage of existing processingresources, be they servers or desktop PC's.

One problem distributed supercomputers run into is that they fall shortwhen it comes to availability, predictability, and maintainability.Distributed supercomputers have problems delivering availability andpredictability because they are typically designed to take advantage ofnon-dedicated resources. The problem is that it is impossible to deliveravailability and predictability when someone else has primary control ofthe resource and your application is simply completing its work when itgets the chance. This makes distributed supercomputers appropriate forsome forms of off-peak processing but not for time-sensitive ormission-critical computing. Finally, setting up, expanding, andmaintaining a distributed supercomputer also requires a significantamount of effort because they tend to offer more of a set of conceptsthan a set of tools. As a result, they require significant amounts ofcustom coding. Again, this reflects the fact that most were developed inthe world of academia where highly trained labor is both cheap andplentiful.

PC's are another option for creating mission-critical applications. PC'shave two clear advantages relative to other solutions. First, PC's arehighly affordable. The relentless progress of Moore's law means thatincreasingly powerful PC's can be acquired for lower and lower prices.The other advantage of PC's is that prices have fallen to such a degreethat many people have begun to regard PC's as disposable. Given how fastthe technology is progressing, in many cases it makes more sense toreplace a PC than to repair it. Of course, the problem with PC's is thatthey do not satisfy the needs of businesses and other organizations whenit comes to scalability, availability, and predictability. First,because PC's were designed to operate as stand-alone machines, they arenot inherently scalable. Instead, the only way to allow them to scale isto link them together into clusters. That can be a very time-consumingprocess. Second, PC's, because they were designed for use byindividuals, were not designed to deliver high levels of availability.As a result, the only way to make a single PC highly available isthrough the use of expensive, custom components. Finally, PC's were notdesigned to handle transaction processing and thus do not have anyprovisions for delivering predictability. The only way to deliver thisfunctionality is to implement it using the operating system or anapplication server. The result is that few organizations even considerusing PC's for mission-critical computing.

In a dynamic environment, it is important to be able to find availableservices. Service Location Protocol, RFC 2165, June 1997, provides onesuch mechanism. The Service Location Protocol provides a scalableframework for the discovery and selection of network services. Usingthis protocol, computers using the Internet no longer need so muchstatic configuration of network services for network based applications.This is especially important as computers become more portable, andusers less tolerant or able to fulfill the demands of network systemadministration. The basic operation in Service Location is that a clientattempts to discover the location of a Service. In smallerinstallations, each service will be configured to respond individuallyto each client. In larger installations, services will register theirservices with one or more Directory Agents, and clients will contact theDirectory Agent to fulfill requests for Service Location information.Clients may discover the whereabouts of a Directory Agent bypreconfiguration, DHCP, or by issuing queries to the Directory AgentDiscovery multicast address.

The following describes the operations a User Agent would employ to findservices on the site's network. The User Agent needs no configuration tobegin network interaction. The User Agent can acquire information toconstruct predicates which describe the services that match the user'sneeds. The User Agent may build on the information received in earliernetwork requests to find the Service Agents advertising serviceinformation.

A User Agent will operate two ways. First, if the User Agent has alreadyobtained the location of a Directory Agent, the User Agent will unicasta request to it in order to resolve a particular request. The DirectoryAgent will unicast a reply to the User Agent. The User Agent will retrya request to a Directory Agent until it gets a reply, so if theDirectory Agent cannot service the request (say it has no information)it must return an response with zero values, possibly with an error codeset.

Second, if the User Agent does not have knowledge of a Directory Agentor if there are no Directory Agents available on the site network, asecond mode of discovery may be used. The User Agent multicasts arequest to the service-specific multicast address, to which the serviceit wishes to locate will respond. All the Service Agents which arelistening to this multicast address will respond, provided they cansatisfy the User Agent's request. A similar mechanism is used forDirectory Agent discovery. Service Agents which have no information forthe User Agent MUST NOT respond.

While the multicast/convergence model may be important for discoveringservices (such as Directory Agents) it is the exception rather than therule. Once a User Agent knows of the location of a Directory Agent, itwill use a unicast request/response transaction. The Service AgentSHOULD listen for multicast requests on the service-specific multicastaddress, and MUST register with an available Directory Agent. ThisDirectory Agent will resolve requests from User Agents which areunicasted using TCP or UDP. This means that a Directory Agent must firstbe discovered, using DHCP, the DA Discovery Multicast address, themulticast mechanism described above, or manual configuration. If theservice is to become unavailable, it should be deregistered with theDirectory Agent. The Directory Agent responds with an acknowledgment toeither a registration or deregistration. Service Registrations include alifetime, and will eventually expire. Service Registrations need to berefreshed by the Service Agent before their Lifetime runs out. If needbe, Service Agents can advertise signed URLs to prove that they areauthorized to provide the service.

New mechanisms for computing are desired, especially those which mayprovide a reliable computing framework and platform, including, but notlimited to those which might produce improved levels of performance andreliability at a much lower cost than that of other solutions.

SUMMARY OF THE INVENTION

A hive of computing engines, typically including clients, requesthandlers, and process handlers, is used to process information.

One embodiment includes a client configured to cause the processing of aprocessing job in order to determine a processing result using one ormore request handlers, and one or more process handlers. The client isconfigured to perform operations including: sending a message assigningthe processing job to a particular request handler, and receiving theprocessing result from the particular request handler. Each of therequest handlers is configured to perform operations including: inresponse to receiving the processing assignment, sending a messageassigning the processing job to a particular process handler; receivingthe processing result from one of the process handlers; and forwardingthe processing result to the client. Each of the process handlers isconfigured to perform operations including: in response to receiving aprocessing assignment of the processing job, processing the processingjob to determine the processing result, and communicating the processingresult to the particular request handler.

In one embodiment, the request handler for a particular job isdetermined using a volunteer pattern initiated by the client. In oneembodiment, the process handler for a particular job is determined usinga volunteer pattern initiated by the request handler currently handlingthe job.

In one embodiment, the request handler is configured to receive stateinformation pertaining to the partial processing of the processing jobsuch that the request handler can assign further processing of theprocessing job to another process handler from a processing stateidentified based on the state information, with such reassignmentpossibly being in response to a detected error condition.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended claims set forth the features of the invention withparticularity. The invention, together with its advantages, may be bestunderstood from the following detailed description taken in conjunctionwith the accompanying drawings of which:

FIG. 1A illustrates an architecture of hives used in one embodiment;

FIG. 1B illustrates a computing platform used for a hive engine forimplementing request handlers, process handlers, and/or other processesof a hive of one embodiment, or also used for simulating the operationof a hive in one embodiment;

FIG. 2A illustrates a hierarchy of a hive, request regions, territories,and processing regions as used in one embodiment;

FIG. 2B illustrates an interaction of a client, request handlers, andprocess handlers of one embodiment;

FIG. 2C illustrates multicast addresses used in one embodiment;

FIG. 2D illustrates the flow of messages between components of oneembodiment;

FIG. 2E illustrates an interaction of a client, request handlers,process handlers and possibly tasks of one embodiment;

FIG. 3 is a flow diagram of a client process used in one embodiment;

FIGS. 4A-C are flow diagrams of request hander processes used in oneembodiment;

FIG. 5A-B are flow diagrams of process hander processes used in oneembodiment;

FIG. 5C is a flow diagram of a task handler process used in oneembodiment;

FIG. 5D is a flow diagram of a recovery layer process used in oneembodiment;

FIG. 6A illustrates a definition of an application used in oneembodiment;

FIG. 6B illustrates a definition of an process flow used in oneembodiment;

FIG. 6C illustrates a process used in one embodiment for executing aprocess flow;

FIG. 7A illustrates a hierarchy of a senior region leaders, regionleaders, and region members among multiple processing regions as used inone embodiment;

FIGS. 7B-7C are flow diagrams of processes used in one embodiment toestablish and maintain a hierarchical relationship among distributedprocesses;

FIG. 8A is a flow diagram of a senior processing region leader processused in one embodiment;

FIG. 8B is a flow diagram of a processing region leader process used inone embodiment;

FIG. 8C illustrates the splitting of a region as performed in oneembodiment; and

FIG. 9 illustrates a process used in one embodiment for initializing ahive engine.

DETAILED DESCRIPTION

A hive of computing engines, typically including request handlers andprocess handlers, is used to process information. Each of the claimsindividually recites an aspect of the invention in its entirety.Moreover, some embodiments described may include, but are not limitedto, inter alia, systems, networks, integrated circuit chips, embeddedprocessors, ASICs, methods, apparatus, and computer-readable mediumcontaining instructions. The embodiments described hereinafter embodyvarious aspects and configurations within the scope and spirit of theinvention, with the figures illustrating exemplary and non-limitingconfigurations.

The term “system” is used generically herein to describe any number ofcomponents, elements, sub-systems, devices, packet switch elements,packet switches, routers, networks, computer and/or communicationdevices or mechanisms, or combinations of components thereof. The term“computer” is used generically herein to describe any number ofcomputers, including, but not limited to personal computers, embeddedprocessing elements and systems, control logic, ASICs, chips,workstations, mainframes, etc. The term “processing element” is usedgenerically herein to describe any type of processing mechanism ordevice, such as a processor, ASIC, field programmable gate array,computer, etc. The term “device” is used generically herein to describeany type of mechanism, including a computer or system or componentthereof. The terms “task” and “process” are used generically herein todescribe any type of running program, including, but not limited to acomputer process, task, thread, executing application, operating system,user process, device driver, native code, machine or other language,etc., and can be interactive and/or non-interactive, executing locallyand/or remotely, executing in foreground and/or background, executing inthe user and/or operating system address spaces, a routine of a libraryand/or standalone application, and is not limited to any particularmemory partitioning technique. The steps, connections, and processing ofsignals and information illustrated in the figures, including, but notlimited to any block and flow diagrams and message sequence charts, maybe performed in the same or in a different serial or parallel orderingand/or by different components and/or processes, threads, etc., and/orover different connections and be combined with other functions in otherembodiments in keeping within the scope and spirit of the invention.Furthermore, the term “identify” is used generically describe any manneror mechanism for directly or indirectly ascertaining something, whichmay included, but is not limited to receiving, retrieving from memory,determining, calculating, generating, etc.

Moreover, the terms “network” and “communications mechanism” are usedgenerically herein to describe one or more networks, communicationsmediums or communications systems, including, but not limited to theInternet, private or public telephone, cellular, wireless, satellite,cable, local area, metropolitan area and/or wide area networks, a cable,electrical connection, bus, etc., and internal communications mechanismssuch as message passing, interprocess communications, shared memory,etc. The term “message” is used generically herein to describe a pieceof information which may or may not be, but is typically communicatedvia one or more communication mechanisms of any type, such as, but notlimited to a packet.

As used herein, the term “packet” refers to packets of all types or anyother units of information or data, including, but not limited to, fixedlength cells and variable length packets, each of which may or may notbe divisible into smaller packets or cells. The term “packet” as usedherein also refers to both the packet itself or a packet indication,such as, but not limited to all or part of a packet or packet header, adata structure value, pointer or index, or any other part oridentification of a packet. Moreover, these packets may contain one ormore types of information, including, but not limited to, voice, data,video, and audio information. The term “item” is used herein to refer toa packet or any other unit or piece of information or data. The phrases“processing a packet” and “packet processing” typically refer toperforming some steps or actions based on the packet, and which may ormay not include modifying and/or forwarding the packet.

The term “storage mechanism” includes any type of memory, storage deviceor other mechanism for maintaining instructions or data in any format.“Computer-readable medium” is an extensible term including any memory,storage device, storage mechanism, etc. The term “memory” includes anyrandom access memory (RAM), read only memory (ROM), flash memory,integrated circuits, and/or other memory components or elements. Theterm “storage device” includes any solid state storage media, diskdrives, diskettes, networked services, tape drives, and other storagedevices. Memories and storage devices may store computer-executableinstructions to be executed by a processing element and/or controllogic, and data which is manipulated by a processing element and/orcontrol logic. The term “data structure” is an extensible term referringto any data element, variable, data structure, data base, and/or one ormore or an organizational schemes that can be applied to data tofacilitate interpreting the data or performing operations on it, suchas, but not limited to memory locations or devices, sets, queues, trees,heaps, lists, linked lists, arrays, tables, pointers, etc. A datastructure is typically maintained in a storage mechanism. The terms“pointer” and “link” are used generically herein to identify somemechanism for referencing or identifying another element, component, orother entity, and these may include, but are not limited to a referenceto a memory or other storage mechanism or location therein, an index ina data structure, a value, etc.

The term “one embodiment” is used herein to reference a particularembodiment, wherein each reference to “one embodiment” may refer to adifferent embodiment, and the use of the term repeatedly herein indescribing associated features, elements and/or limitations does notestablish a cumulative set of associated features, elements and/orlimitations that each and every embodiment must include, although anembodiment typically may include all these features, elements and/orlimitations.

In addition, the terms “first,” “second,” etc. are typically used hereinto denote different units (e.g., a first element, a second element). Theuse of these terms herein does not necessarily connote an ordering suchas one unit or event occurring or coming before the another, but ratherprovides a mechanism to distinguish between particular units.Additionally, the use of a singular tense of a noun is non-limiting,with its use typically including one or more of the particular itemrather than just one (e.g., the use of the word “memory” typicallyrefers to one or more memories without having to specify “memory ormemories,” or “one or more memories” or “at least one memory”, etc.)Moreover, the phrases “based on x” and “in response to x” are used toindicate a minimum set of items x from which something is derived orcaused, wherein “x” is extensible and does not necessarily describe acomplete list of items on which the operation is performed, etc.Additionally, the phrase “coupled to” is used to indicate some level ofdirect or indirect connection between two elements or devices, with thecoupling device or devices modify or not modifying the coupled signal orcommunicated information. The term “subset” is used to, indicate a groupof all or less than all of the elements of a set. Moreover, the term“or” is used herein to identify a selection of one or more, includingall, of the conjunctive items.

Numerous means for processing information using a hive of computing/hiveengines are disclosed. One implementation includes a request regionincluding multiple request handlers and multiple processing regions,each typically including multiple process handlers. Each request handleris configured to respond to a client service request of a processingjob, and if identified to handle the processing job: to query one ormore of the processing regions to identify and assign a particularprocess handler to service the processing job, and to receive aprocessing result from the particular process handler. As typically usedherein, a result corresponds to the outcome of a successfully orunsuccessfully completed job, task or other operation or an errorcondition, and typically includes one or more indications of a finalvalue or outcome and/or state information (e.g., indications of to theprocessing performed or not performed, partial or final results, errordescriptors, etc.) Each of the process handlers is configured to respondto such a query, and if identified as the particular process handler: toservice the processing job, to process the processing job, to updatesaid identified request handler with state information pertaining topartial processing of said processing job, and to communicate theprocessing result to the identified request handler.

In one embodiment, a volunteer pattern allows a software application(e.g., client process, request handler, process handler, task handler,tasks, or another hive engine process, etc.) to automatically detect agroup of software applications on the same network, and to select andcommunicate with the most appropriate application without any priorknowledge to the location and capabilities of the chosen softwareapplication. In one embodiment, messages are sent among processestypically using multicast UDP, unicast UDP, and standard TCPconnections.

In one embodiment, the volunteer pattern includes the following steps.First, hive engines that wish to volunteer its capabilities begin bylistening for volunteer requests on a known multicast address. Next, aclient looking for a request handler to handle its request transmits itsneeds by issuing a volunteer or service request packet. The servicerequest packet is a small text buffer which includes the type of serviceit is requesting and any potential parameters of that request. Theservice request packet also includes the return IP address of the clientfor hive engines to use to communicate their volunteer responses. Thevolunteer packet is communicated via multicast to the known multicastgroup corresponding to the request region. Request handlers of multiplehive engines on the client's network will detect this request. Third,hive engines that receive the service request packet examine itscontents. If the hive engine is capable of servicing this request, itresponds by sending a response (e.g., a UDP packet) to the client whichmade the request. The UDP packet typically contains the TCP address ofthe hive engine's communication port. Unicast UDP packets are used sothat only the client that initiated the service request will receive thevolunteer responses from the request handlers. Fourth, the clientreceives unicast UDP packets from the hive engines, selects one, andconnects to the hive engine via TCP socket. The client and hive enginewill typically use this socket for all subsequent communications duringthe processing of this application.

In one embodiment, regionalization is used to allow participating hiveengines on the same network to detect each other and organize intological groups of processing regions without any prior configuration tominimize bandwidth usage and CPU consumption in the entire system.Regionalization provides an automated mechanism that allows theseprocessing regions grow and split as needed, which may provide for anunlimited growth of a hive. Thus, volunteer requests (e.g., processingrequests, task requests, etc.) can be within a processing region withoutaffecting all hive engines sending these requests or othercommunications using a multicast address assigned to a specificprocessing region. This places a bound on the number of responses to begenerated (e.g., by the number of hive engines in a processing region.)Typically, hive engines participate in an automated self-organizationmechanisms, which allows participating hive engines on the same local orwide area network to detect each other and organize into logical groupswithout any prior configuration. However, an embodiment may use anymechanism for defining a regionalization, or even one embodiment doesnot use regionalization. For example, in one embodiment, a hive engineis pre-configured with parameters to define which region or regions inwhich to participate; while in one embodiment, users or a centralizedcontrol system is used to specify to one or more hive engines whichregion or regions in which to participate.

A hive typically has multiple processing regions and a single requestregion; although, one embodiment includes multiple request regions andone or more processing regions. One way to view a processing region isthat it is a set of processes on one or more hive engines for executingprocessing jobs. In one embodiment, a processing region has a leaderthat keeps track of the number of hive engines in the region. If thenumber of hive engines in the region reaches the user defined maximum,the region leader instructs the hive engines in the region to divideinto two separate smaller regions. If the number of hive engines in theregions reaches the user defined minimum, the region leader instructsthe hive engines in the region to join other regions in the hive.

In one embodiment, the processing regions are self-healing in that ifthe region leader shuts down for any reason all the region membersdetect the lack of a region leader. A region member promotes itself toregion leader. If a processing region has multiple region leaders, theyoungest region leaders demotes themselves back to region members,leaving one region leader.

A request region typically hides that the hive consists of multipleregions and directs the processing load across all the regions. From oneperspective, spreading the request region across multiple hive enginesprovides an increased level of fault tolerance, as these services detectthe loss of a connection and rebuild or shutdown as necessary. The hiverecovers most failure cases, however, when a request is in anindeterminate state, the request is typically terminated to preventmultiple executions.

In one embodiment, a single senior region leader forms the requestregion. The senior region leader discovers the region leaders via thevolunteer pattern. The senior region leader discovers the size of therequest region by asking the region leaders for the number of hiveengines in their region that are also members of the request region. Ifthe request region has too many or too few members, the senior regionleader directs the region leaders to re-allocate the hive engines to orfrom the request region. The request region is typically self-healing inthat if the senior region leader shuts down for any reason all theregion leaders detect the lack of a senior region leader. A regionleader promotes itself to senior region leader. If the new senior regionleader is not the most senior region leader, the senior region leaderdemotes itself and the most senior region leader promotes itself tosenior region leader. If more than one senior region leader exists, thesenior region leaders that are less senior or junior to another seniorregion leader demotes itself.

In one embodiment, a client processing job is specified in terms of aprocess flow, typically specifying a set of tasks as well statevariables typically before and after each task for storing stateinformation. The hive process flow contains the information on thesequence of sub-routines to be called, timeout and retry information ifthe sub-routines fail, and which sub-routine to call next based on thesub-routine's result. Once specified, it is up to the hive software toexecute the sub-routines in the process flow. A process flow maydescribed in any manner or format. For example, in one embodiment, aprocess flow is described in a XML process definition file. The processflow definition file defines the process flow name, the task to beperformed, the task's recovery procedure including the timeout limit andretry limit, and the transition from one state to the next state basedon the previous task's result.

In order to maintain high-availability and fault tolerance, a clientprocessing job is typically performed using a self-organized,non-administered, network of services across several hive engines thatwork together to guarantee execution of a request even in the event thatany of the individual services or hive engines fail. For example, in oneembodiment, a processing job is received by a request handler from aclient using the volunteer pattern. The request engine selects a processhandler based on pattern. The process handler proceeds to perform theprocessing job, and at intermediate steps within the process flow, theprocess handler communicates state information to the request engine,such that the state and progress of the processing job at discrete stepsis known by multiple processes, typically on different physical hiveengines, and possibly in different territories (which may be defined tobe in physically different locations, or using different communicationsand/or electrical systems, etc.) Thus, should a failure occur, theprocessing job typically can be resumed by another process handler newlyselected by the request handler, or possibly completed by the originalprocess handler with it storing results and/or communicating the resultsto the client via a different path (e.g., using a different requesthandler, etc.)

In one embodiment, processing a request typically includes the requestsetup, request processing, and request teardown. In the request setup,the client submits a request for a volunteer to the request region. Arequest handler receives the request, opens a TCP connection, and sendsa response to the client. The client sends the request over the TCPconnection to the request handler. The request handler receives therequest and submits a request for a volunteer. A process handlerreceives the request, opens a TCP connection, and sends a response tothe request handler. The request handler receives the response and sendsthe request over the TCP connection to the process handler. The processhandler receives the request and sends an acknowledgement message. Therequest handler receives the acknowledgement message then sends anacknowledgement message to the client. The client receives theacknowledgement message then sends a process command to the requesthandler. The request handler receives the process command sends theprocess command to the process handler. The process handler receives theprocess command and begins processing the request. If the client losesconnection with the request handler during this procedure, the clientshould perform a retry.

In one embodiment, in the request process procedure, the process handlersubmits a volunteer request to a processing region. A task handlerreceives the volunteer request, opens a TCP connection, and sends aresponse. The process handler receives the volunteer response and sendsthe first task in the process flow to the task handler over the TCPconnection. The task handler processes the task and sends the results tothe process handler. If the task does not complete within the specifiedamount of time and retries are set to zero, the request handler returnsan error code as the final result to the request handler. If the taskdoes not complete within the specified amount of time and retries aregreater than zero, the request handler resubmits the task to anothertask handler. If snapshot is enabled on this task or if retries is setto zero, the process handler sends the result to the request handler.This repeats until the next state is finish. When the next state isfinish, the process handler sends the final result to the requesthandler. If the client loses connection with the request handler duringthis procedure, the client should perform a recover.

In one embodiment, in the request teardown procedure, the requesthandler sends the final result to the client. The client receives theresult and sends an acknowledgement to the request handler. The requesthandler receives the acknowledgement and sends an acknowledgement to theprocess handler. If the client loses connection with the request handlerduring this procedure, the client should perform a recover.

In one embodiment, the task service runs on each worker machine. Taskservices have an IP address and assigned TCP port on their workermachine. All task services in the Hive share common UDP multicast groupsbased on their worker machine's current region. On completion of thevolunteer pattern for a simple task, the connected TCP socket will bepassed off to the task handler. When responding to a volunteer patternfor a daemon task, this service will UDP the daemon task's IP and portto the requester. The service has both task handlers and daemon tasks.Upon receiving a task to execute from a process handler, the servicewill spin off a task handler or delegate the task to a daemon task, asappropriate. Upon completion of the task, the task handler or daemontask will return the results to the process handler.

One embodiment uses an intra-process recovery which enables the hive torecover from a connection loss between the client and the requesthandler while the request handler is overseeing the processing of arequest. When the client loses the connection with a first requesthandler, once the request processing has completed the request setupphase, the first request handler continues processing the request andthe client submits a request for a new request handler (second requesthandler). The client issues the recover command and second requesthandler listens queries the recover service for a user-defined amount oftime. If second request handler does not receive the result within thespecified amount of time, second request handler returns an error. Whenfirst request handler receives the final result, first request handlerwrites the final result to the recover service.

One embodiment operates slightly differently as multiple processhandlers are used for each step in a process flow. For example, bothprocess handlers typically maintain the current state of the requestsuch that if either of the process handlers is lost, the other picks upin its place. If the request handler is lost, the client and/or processhandlers can establish a new request handler. The request handlermanages the interface between software requesting processing from thehive and the hive. A primary process handler is a service that walks arequest through the steps and recovery defined in a process flow. Asecondary process handler is a service that monitors the primary processhandler. If something happens to the primary process handler, thesecondary process handler continues going through the steps and recoverydefined in a process flow. A task handler is a service that performs thesub-routine defined in the process flow.

For example, in one embodiment, first, a request handler finds twoprocess handlers. The request handler designates one as the primaryprocess handler and the other as the secondary process handler. Next,the request handler sends the primary process handler the secondaryprocess handler's IP address and sends the secondary process handler theprimary process handler's IP address. The primary process handler andsecondary process handler open a TCP port for communication then sendacknowledgement messages to the request handler. The primary processhandler finds a task handler. The task handler opens a TCP port andsends the request to the primary process handler. The primary processhandler prepares the initial process flow state and sends that state tothe secondary process handler. The secondary process handler and therequest handler monitor the task states over the TCP connection. Thetask handler processes the request, sends the result to the primaryprocess handler.

One embodiment provides an assimilation mechanism which recognizes newhive engines trying to join a hive. These steps occur without stoppingexecution of the entire hive, and he hive updates its hive engines in ameasured rate to ensure that portions of the hive are continuallyprocessing requests ensuring constant availability of the hiveapplications.

In one embodiment, when a new hive engine joins the hive, the new hiveengine finds the operating system image and the base hive software viaDHCP. The new hive engine self installs the OS image and hive softwareusing automated scripts defined by client. If a hive engine has an oldversion of the OS, the region leader makes the hive engine unavailablefor processing. The hive engine is erased and rebooted. The hive enginethen joins the hive as a new hive engine and re-installs the OS and hivesoftware accordingly.

In addition, in one embodiment, when a hive engine joins the hive, thehive engine sends a request to the region leader. The hive enginereceives a response from the region leader and selects a region to join.The region leader queries the hive engine for information aboutservices, software, and versions. If the region leader is running anewer version of the hive system, the region leader makes the hiveengine unavailable for processing. The region leader updates the hiveengine by transmitting the current version of the hive system. The hiveengine installs the update and commences processing. If the hive engineis running a newer version of hive system than the region leader, theregion leader makes itself unavailable for process, receives the newerversion of the hive system from the hive engine, installs the software,and continues processing. Once the region leader is updated, the regionleader begins updating its region's members and the other regionleaders. For example, in one embodiment, a hive engine then receives aresponse from the region leaders and selects a region to join. Theregion leader queries the hive engine for information about services,software, and versions. If the region leader is running the most currentversion of the hive applications, the region leader automaticallyupdates the hive engine's hive applications. If the hive engine isrunning the most current version of the hive applications, the regionleader automatically updates its hive applications. Once the regionleader is updated, the region leader begins updating its region'smembers and the other region leaders.

Turning to the figures, FIG. 1A illustrates an architecture of hivesused in one embodiment. Shown are multiple hives 100-101. A hive 100-101is a logical grouping of one or more hive engines (e.g., computers orother computing devices) networked together to perform processingresources to one or more hive clients 110. For example, hive 100includes multiple hive engines 105-106 connected over a network (or anycommunication mechanism) 107.

In one embodiment, a hive is a decentralized network of commodityhardware working cooperatively to provide vast computing power. A hivetypically provides high-availability, high-scalability, low-maintenance,and predictable-time computations to applications (e.g., thosecorresponding to processing jobs of clients) executed in the hive. Eachhive engine in the hive is typically capable to individually deploy andexecute hive applications. When placed on the same network, hive enginesseek each other out to pool resources and to add availability andscalability.

FIG. 1B illustrates a computing platform used for a hive engine forimplementing request handlers, process handlers, and/or other processesof a hive as used in one embodiment (or also used for simulating theoperation of one or more elements of a hive in one embodiment). Asshown, hive engine 120 is configured to execute request handlers,process handler, and other hive processes, and to communicate withclients and other hive engines as discussed herein.

In one embodiment, hive engine 120 includes a processing element 121,memory 122, storage devices 123, communications/network interface 124,and possibly resources/interfaces (i.e., to communicate to otherresources ) which may be required for a particular hive application(e.g., specialized hardware, databases, I/O devices, or any otherdevice, etc.) Elements 121-125 are typically coupled via one or morecommunications mechanisms 129 (shown as a bus for illustrativepurposes). Various embodiments of hive engine 120 may include more orless elements. The operation of hive engine 120 is typically controlledby processing element 121 using memory 122 and storage devices 123 toperform one or more hive processes, hive tasks, or other hive operationsaccording to the invention. Memory 122 is one type of computer-readablemedium, and typically comprises random access memory (RAM), read onlymemory (ROM), flash memory, integrated circuits, and/or other memorycomponents. Memory 122 typically stores computer-executable instructionsto be executed by processing element 121 and/or data which ismanipulated by processing element 121 for implementing functionality inaccordance with the invention. Storage devices 123 are another type ofcomputer-readable medium, and typically comprise solid state storagemedia, disk drives, diskettes, networked services, tape drives, andother storage devices. Storage devices 123 typically storecomputer-executable instructions to be executed by processing element121 and/or data which is manipulated by processing element 121 forimplementing functionality in accordance with the invention.

In one embodiment, hive engine 120 is used as a simulation engine 120 tosimulate one or more hive engines, and/or one or more hive processes,tasks, or other hive functions, such as, but not limited to thosedisclosed herein, especially the operations, methods, steps andcommunication of messages illustrated by the block and flow diagrams andmessages sequence charts. Hive simulator engine 120 typically is used tosimulate the performance and availability of hive application fabrics.The simulator allows dynamic simulation of any environment using simpletext directives or a graphical user interface. For example, hivesimulator engine 120 can be used to determine the hive performance usingparticular computing hardware by specifying such things as the computertype, instantiation parameters, and connection fabric, which is used byhive simulator engine 120 to produce a representation of the performanceof a corresponding hive. In one embodiment, multiple hive simulatorengines 120 are used, such as a unique three-level, two-dimensional modeconnection fabric that allows hive simulator engines 120 to transmitrequests uni-directionally or bi-directionally and to access other hivesimulator engines 120 for subset processing while processing a request.Thus, one or more hive simulator engines 120 allow for modeling at thesoftware level, hardware level, or both levels. Additionally, a hivesimulator engine 120 is typically able to transmit requests through asimulated network or real hive network, such as hive 100 (FIG. 1A).

FIG. 2A illustrates a hierarchy of a hive, request regions, territories,and processing regions as used in one embodiment. As shown, hive 200 islogically divided into one or more request regions 205 (although mosthives use only one request regions), territories 210 and 216, withmultiple processing regions 211-212 and 217-218.

The use of territories 210 and 216 provides a mechanism for associatinga physical location or quality of a corresponding hive engine which canbe used, for example, in determining which responding request or processhandlers to select via a volunteer pattern. When defined based onphysical location, if performance is the major issue, then it istypically advantageous (but not required) to process all requests withinthe same territory. If reliability is the major issue, then it istypically advantageous (but not required) store state recoverinformation in another territory.

FIG. 2B illustrates an interaction of a client, request handlers, andprocess handlers of one embodiment. Client 220 generates a servicerequest 221 to request handlers 222, such as via a request regionmulticast message, one or more messages, a broadcast message, or othercommunication mechanisms. Those request handlers 222 that are availableto process the request return responses 223 to client 220, typically viaa unicast message directly to client 220 which includes a communicationsport to use should the sending request handler be selected by client220. Client 220 selects, optionally based on territory considerations,typically one (but possibly more) of the responding request handlers,and communicates processing job 224 to the selected request handler 225.

In response, selected request handler 225 generates a processing request226 to process handlers 227, such a via one or more processing regionmulticast messages or other communication mechanisms. Those processhandlers 227 that are available to process the request return responses228 to selected request handler 225, typically via a unicast messagedirectly to selected request handler 225 which includes a communicationsport to use should the sending request handler be selected by selectedrequest handler 225. Selected request handler 225 selects, optionallybased on territory considerations, typically one (but possibly more) ofthe responding process handlers, and communicates processing job withstate information 229 to the selected process handler 230. Inclusion ofthe state information is emphasized in regards to processing job withstate information 229 because the processing job might be ran from thebeginning or initialization state, or from an intermittent position orstate, such as might happen in response to an error or timeoutcondition.

In response, selected process handler 230 proceeds to execute theprocess flow (or any other specified application), and at defined pointsin the process flow, updates selected request handler 225 withupdated/progressive state information 237. Typically based on theprocess flow, selected process handler 230 will sequentially (althoughone embodiment allows for multiple tasks or sub-processes to be executedin parallel) cause the tasks or processing requests to be performedwithin the same hive engine or by other hive engines.

In one embodiment, selected process handler 230 selects a hive engine toperform a particular task using a volunteer pattern. For example,selected process handler 230 sends a multicast task request 231 to taskhandlers typically within the processing region (although oneembodiment, sends task requests 231 to hive engines in one or moreprocessing and/or request regions). Those task handlers 232 able toperform the corresponding task send a response message 233 to selectedprocess handler 230, which selects, possibly based on territory, hiveengine (e.g., itself as less overhead is incurred to perform the taskwithin the same hive engine) or other considerations, one of theresponding task handlers 232. Selected process handler 230 theninitiates the task and communicates state information via message 234 tothe selected task handler 235, which performs the task and returns stateinformation 236 to selected process handler 230. If there are more tasksto perform, selected process handler 230 typically then repeats thisprocess such that tasks within a process flow or application may or maynot be performed by different hive engines. Upon completion of theapplication/process flow, selected process handler 230 forwards thefinal state information (e.g., the result) 237 to selected requesthandler 225, which in turn, forwards the result and/or other information238 to client 220.

In one embodiment, selected process handler 230 performs tasks itself orcauses tasks to be performed within the hive engine in which it resides(and thus selected task handler 235 is within this hive engine, and oneembodiment does not send task request message 231 or it is sentinternally within the hive engine.) In one embodiment, selected taskhandler 235 is a separate process or thread running in the same hiveengine as selected process handler 230. Upon completion of theapplication/process flow, selected process handler 230 forwards thefinal state information (e.g., the result) 237 to selected requesthandler 225, which in turn, forwards the result and/or other information238 to client 220.

FIG. 2C illustrates multicast addresses 240 used in one embodiment. Asshown, multicasts addresses 240 includes: a multicast request regionaddress 241 using which a client typically sends a service requestmessage, a processing region leader intercommunication multicast address242 used for processing region leaders to communicate among themselves,a processing region active region indications multicast address 243which is typically used to periodically send-out messages by regionleaders to indicate which processing regions are currently active, andmultiple processing region multicasts addresses 244, one typically foreach processing region of the hive. Of course, different sets orconfigurations of multicast addresses or even different communicationsmechanisms may be used in one embodiment within the scope and spirit ofthe invention.

FIG. 2D illustrates the flow of messages among components of oneembodiment. Client 250 sends a multicast hive service request message256 into the request region 251 of the hive. Request handlers availablefor performing the application corresponding to request 256 respond withUDP messages 257 to client 250, which selects selected request handler252, one of the responding request handlers. In one embodiment, thisselection is performed based on territory or other considerations, oreven on a random basis. Client 250 then communicates the processing jobin a message 258 over a TCP connection to the selected request handler252.

In response and using a similar volunteer pattern, selected requesthandler 252 multicasts a processing request message 259 to a selectedprocessing region 253, and receives UDP response messages 260 fromavailable processing engines to service the request (e.g., perform theprocessing job). Selected request handler 252 selects selected processhandler 254, one of the responding request handlers. In one embodiment,this selection is performed based on territory or other considerations,or even on a random basis. Selected request handler 252 then forwardsthe processing job with state information in message 261 to selectedprocess handler 254, which returns an acknowledgement message 262. Inresponse, selected request handler 252 sends an acknowledgement message263 to client 250 (e.g., so that it knows that the processing is aboutto be performed.)

Selected process handler 254 then causes the processing job to beexecuted, typically by performing tasks within the same hive engine ifpossible for optimization reasons, or by sending out one or more tasks(possibly using a volunteer pattern) to other hive engines. Thus,selected process handler 254 optionally sends a multicast task requestmessage 264 typically within it own processing region (i.e., selectedprocessing region 253) (and/or optionally to one or more otherprocessing or request regions), and receives responses 265 indicatingavailable task handlers for processing the corresponding task. Taskrequest message 264 typically includes an indication of the type or nameof the task or task processing to be performed so that taskhandlers/hive engines can use this information to determine whether theycan perform the task, and if not, they typically do not send a responsemessage 265 (as it is less overhead than sending a response messageindicating the corresponding task handler/hive engine cannot perform thetask.) Note, in one embodiment, a task handler within the same hiveengine as selected process handler 254 sends a response message 265.

Whether a task handler to perform the first task is explicitly orimplicitly determined, selected process handler initiates a first task266, which is performed by one of one or more individual task threads255 (which may be the same or different task threads on the same ordifferent hive engines), which upon completion (whether naturally orbecause of an error or timeout condition), returns state information 272to selected process handler 254, which in turn updates selected requesthandler 252 via progressive state message 273. (Note, if there was onlyone task, then completion/state message 276 would have been sent inresponse to completion of the task.) This may continue for multipletasks as indicated by optional MCAST task request and response messages268-269 and task-n initiation 270 and state messages 272. Whenprocessing of the application/process flow is completed as determined byselected process handler 254 in response to state messages from theindividual task threads 255, selected process handler 254 forwards acompletion and result state information 276 to selected process handler252, which forwards a result message 277 to client 250. In response,client 250 sends an acknowledgement message 278 to confirm receipt ofthe result (indicating error recovery operations do not need to beperformed), and an acknowledgement message 279 is forwarded to selectedprocess handler 254, and processing of the processing job is complete.

FIG. 2E illustrates an interaction of a client, request handlers,process handlers and possibly tasks of one embodiment. Many of theprocesses and much of the flow of information is the same as illustratedin FIG. 2B and described herein, and thus will not be repeated. FIG. 2Eis used to emphasize and explicitly illustrate that differentembodiments may implement features differently, and to emphasize that aprocess flow may specify tasks or even other process flows to beperformed or the same process flow to be performed recursively.

For example, as shown, selected process handler 230 of FIG. 2B isreplaced with selected process handler 280 in FIG. 2E. Selected processhandler 280, in response to being assigned to execute the clientsprocessing job by receiving processing job with state informationmessage 229, proceeds to execute the corresponding application/processflow, which may optionally include performing a volunteer pattern usingprocessing or task request messages 281 and response messages 283to/from one or more task or process handlers 282. In response to thevolunteer operation or directly in response to receiving the processingjob with state information message 229, selected process handler 280will sequentially (although one embodiment allows for multiple tasks orsub-processes to be executed in parallel) perform itself or send outtasks or processing requests to corresponding selected task or processhandlers 290, in which case task or processing job with stateinformation messages 284 are typically sent and results or stateinformation messages 296 are typically received. The number of levelsused in performing a processing job is unbounded as indicated in FIG.2E.

FIG. 3 is a flow diagram of a client process used in one embodiment.Processing begins with process block 300, and proceeds to process block302, wherein an application, data, and hive to process these isidentified. Next, in process block 304, a multicast service requestmessage indicating application is sent into the request layer of theselected hive. In process block 306, responses are received from thehive (if no responses are received, processing returns to process block302 or 304 in one embodiment). Next, in process block 308, a requesthandler is selected based on the responses, and a communicationsconnection is established to the selected request handler in processblock 310. Next, in process block 312, the processing job is submittedto the selected request handler and a global unique identifier (GUID) isincluded so that the client and hive can uniquely identify theparticular processing job. As determined in process block 314, if anacknowledgement message is not received from the hive indicating the jobis being processed within a timeframe, then processing returns toprocess block 304.

Otherwise, if results are received from the hive within the requisitetimeframe as determined in process block 320, then an acknowledgementmessage is returned to the hive in process block 322, and processing iscomplete as indicated by process block 324. Otherwise, as determined inprocess block 330, if the client determines it wishes to perform arecover operation, then in process block 332, a multicast recoveryrequest message specifying the GUID is sent to the request layer of thehive, and processing returns to process block 320 to await the recoveryresults. Otherwise, as determined in process block 340, if the clientdetermines to again request the job be performed, then processingreturns to process block 304. Otherwise, local error processing isoptionally performed in process block 342, and processing is complete asindicated by process block 344.

FIGS. 4A-C are flow diagrams of request hander processes used in oneembodiment. FIG. 4A illustrates a process used in one embodiment forresponding to service requests of clients. Processing begins withprocess block 400, and proceeds to process block 402, wherein amulticast port is opened for receiving service request messages. Asdetermined in process blocks 404 and 406, until a service request isreceived and the request handler is available to handle the request,processing returns to process block 404. Otherwise, the request handlerresponds in process block 408 by sending a response message to therequesting client, with the response message typically identifying aport to use and the GUID of the received service request. As determinedin process block 410, if the service request corresponds to a recoveryrequest, then in process block 412, a recovery thread is initialized(such as that corresponding to the flow diagram of FIG. 4C) or therecovery operation is directly performed. Otherwise, in process block414, a selected request handler thread is initialized (such as thatcorresponding to the flow diagram of FIG. 4B) or the request is handleddirectly. Processing returns to process block 404 to respond to morerequests.

FIG. 4B illustrates a flow diagram of a process used by a selectedrequest handler in one embodiment. Processing begins with process block430, and loops between process blocks 432 and 434 until a job isreceived (and then processing proceeds to process block 440) or until atimeout condition is detected and in which case, processing is completeas indicated by process block 436.

After a processing job has been received (e.g., this process has beenselected by the client to handle the request), a state data structure isinitialized in process block 440. Then, in process block 442, amulticast processing request message is sent into one of the processinglayers of the hive. As determined in process block 444, if no responsesare received within a requisite timeframe, then a no processing handlerresponse message is returned to the client in process block 445, andprocessing is complete as indicated by process block 436.

Otherwise, in process block 446, a particular process handler isselected. In one embodiment, this selection is performed based onterritories (e.g., a process handler in a different territory than theselected request handler), other considerations or even on a randombasis. In process block 448, a communications connection is establishedif necessary to the selected process handler, and the state informationand data for the client processing request is sent (which may correspondto the initial state of the data received from the client or to anintermediate state of processing the client job request).

As determined in process block 450, if an error or timeout condition isdetected, processing returns to process block 442. Otherwise, asdetermined in process block 452, until a state update message isreceived, processing returns to process block 450. As determined inprocess block 454, if the received state is not the finished orcompleted state, then in process block 456, the state data structure isupdated, and processing returns to process block 450. Otherwise,processing has been completed, and in process block 458, the result iscommunicated to the client; in process block 460, the communicationsconnection is closed; and processing is complete as indicated by processblock 462.

FIG. 4C illustrates a flow diagram of a process used by a selectedrequest handler performing error recovery in one embodiment. Processingbegins with process block 470, and loops between process blocks 472 and474 until a job is received (and then processing proceeds to processblock 478) or until a timeout condition is detected and in which case,processing is complete as indicated by process block 476.

After a processing job has been received (e.g., this process has beenselected by the client to perform the recover processing), in processblock 478, a multicast recovery request message specifying the GUID ofthe job being recovered is sent into one or more of the recovery modulesof the hive. As determined in process block 480, if no responses arereceived within a requisite timeframe, then a no recover responsemessage is returned to the client in process block 481, and processingis complete as indicated by process block 476.

Otherwise, in process block 482, a particular recovery handler isselected, possibly based on territory considerations—such as a recoveryhandler in a different territory then this selected request handler. Inprocess block 484, a communications connection is established ifnecessary to the selected recovery handler thread, and a recoveryrequest is sent, typically including the GUID or other indication of thejob to be recovered.

As determined in process block 486, if an error or timeout condition isdetected, processing returns to process block 478. Otherwise, therecovered information is received as indicated by process block 488. Inprocess block 490, the information is typically communicated to theclient, or if this communication fails, it is saved to the recoverysystem. In one embodiment, the partially completed state, errors and/orother indications are stored to a local storage mechanism (e.g., somecomputer-readable medium) to be made available for use by a recoveryprocess. In one embodiment, more significant process handling isperformed, or the error communicating the error to another process,thread or hive engine for handling. The communications connection isthen closed in process block 492, and processing is complete asindicated by process block 494.

FIGS. 5A-B are flow diagrams of process hander processes used in oneembodiment. FIG. 5A illustrates a process used in one embodiment forresponding to service requests of request handlers. Processing beginswith process block 500, and proceeds to process block 502, wherein amulticast port is opened for receiving processing request messages. Asdetermined in process blocks 504 and 506, until a processing request isreceived and the process handler is available to handle the request,processing returns to process block 504. Otherwise, the process handlerresponds in process block 508 by sending a response message to therequesting request handler, with the response message typicallyidentifying a port to use and possibly the GUID corresponding to thereceived processing request. The processing request is received inprocess block 510. Next, in process block 512, a selected processhandler thread is initialized (such as that corresponding to the flowdiagram of FIG. 5B) or the processing request is handled directly.Processing returns to process block 504 to respond to more requests.

FIG. 5B illustrates a flow diagram of a process used by a selectedprocess handler in one embodiment. Processing begins with process block520, and loops between process blocks 522 and 524 until a job isreceived (and then processing proceeds to process block 530) or until atimeout condition is detected and in which case, processing is completeas indicated by process block 526.

After a processing job has been received (e.g., this process has beenselected by a selected request handler (or possibly other processhandler) to handle the request), a state data structure is initializedin process block 530. In process block 532, the processing requirementsof the next statement(s) within the process flow corresponding to thereceived job are identified. As determined in process block 534, if asub-process is to be spawned (e.g., the process flow specifies a processflow to be executed), then in process block 536, the current state ispushed on to a state stack and the state is initialized to that of thenew process flow, the selected request handler is updated in processblock 538, and processing returns to process block 532 to process thenew process flow.

Otherwise, as determined in process block 540, if the task handler isnot already known (e.g., an optimization to perform the task on the samehive engine) such as it is not guaranteed to be performed locally, thetask is a “limited task” in that it can only be performed by a subset ofthe task handlers or the processing of the task is made available toother hive engines (e.g., for performance or load balancing etc.), thenin process block 542 the task handler to perform the task is identified.One embodiment identifies the task handler by sending a multicast taskrequest messages, receives the responses, and selects, based onterritory, load or other considerations, a task handler to perform thetask.

Limited tasks provide a mechanism for identifying hive engines that havespecial hardware or other resources. Task handlers only on the hiveengines with the specialized hardware or other resources possiblyrequired to perform the task will be enabled to perform thecorresponding task and thus these enabled task handlers will be the onesto respond to a task request for the corresponding task. Additionally,limited tasks provide a mechanism to limit the number of task handlersor hive engines allowed to access a particular resource by restrictingthe number and/or location of task handlers allowed to perform a taskthat accesses the particular resource. Thus, limited tasks may be usefulto limit the rate or number of accesses to a particular resource (e.g.,database engine, a storage device, a printer, etc.)

In process block 544, a task is initiated to perform the next operationidentified in the current process flow with the current stateinformation and characteristics (e.g., timeout, number of retries, etc.)on the identified, selected, or already known task handler. Asdetermined in process block 546, after completion of the processingrequirements of the processing statement(s), if the finish state has notbeen reached, then the state data structure is updated with the taskresult in process block 548, the selected request handler is updatedwith the current state information in process block 549, and processingreturns to process block 532.

Otherwise, processing is completed of the current process flow asdetermined in process block 546, and if the current process flow is asub-process (e.g., spawned process flow) (as determined in process block550), then in process block 552, the state is popped from the statestack, and processing proceeds to process block 548. Otherwise, inprocess block 554, the result/state information is communicated to theselected request hander. As determined in process block 555, if an errorhas been detected, then error processing is performed in process block556. In process block 558, the communications connection is closed, andprocessing is complete as indicated by process block 559. Note, in someembodiments, communications connections are not established anddisconnected each time, but rather a same communications channel is usedmore than once.

FIG. 5C illustrates a flow diagram of a task handler performed by a hiveengine in one embodiment. Processing begins with process block 580. Asdetermined in process blocks 581 and 583, until a task request isreceived and the task handler is available to handle the request,processing returns to process block 581. Otherwise, the task handlerresponds in process block 584 by sending a response message to therequesting process (typically a process handler), with the responsemessage typically identifying a port to use and the GUID of the receivedtask request. As determined in process block 585, if the task isactually received (e.g., this task handler was selected by the processhandler sending the task request), then in process block 586, the taskis performed or at least attempted to be performed and resultant stateinformation (e.g., completed state, partially completed state, errorsand/or other indications) sent to the requesting process handler.Processing returns to process block 581. Note, in one embodiment,multiple processes illustrated in process block 5C or some variantthereof are performed simultaneously by a hive engine for responding tomultiple task requests and/or performing tasks in parallel.

FIG. 5D illustrates a flow diagram of a recovery processing performed bya hive engine in one embodiment. Processing begins with process block590, and loops between process blocks 591 and 592 until a recovery jobis received (and then processing proceeds to process block 594) or untila timeout condition is detected and in which case, processing iscomplete as indicated by process block 593. In process block 594, therecovery is retrieved from local storage and is communicated to theselected request hander. As determined in process block 595, if an errorhas been detected, then error processing is performed in process block595. In process block 598, the communications connection is closed, andprocessing is complete as indicated by process block 599.

In one embodiment, a hive application is a collection of process flowsthat carry out specific sets of tasks. Applications can share processflows. An application definition file (XML descriptor file) typicallydescribes the application, and the application definition file typicallyconsists of the following: application name, process flow names, tasknames and module file names, support files, and/or configuration filenames.

FIG. 6A illustrates an example definition file 600 of an application foruse in one embodiment. As show, application definition file 600specifies a set of corresponding process flows 601, tasks 602, supportfiles 603, and configuration files 604.

FIG. 6B illustrates a definition of an process flow 620 “doProcessOne”used in one embodiment. Shown are four process flow statements 621-624,each specifying its beginning state, tasks to be performed, and nextstate depending on the outcome of the statements execution.

FIG. 6C illustrates a process used in one embodiment for executing aprocess flow or processing job, such as that illustrated in FIG. 6B.Note, in one embodiment, the process illustrated in FIG. 5B is used toexecute a process flow or processing job. In one embodiment, acombination of the processes illustrated in FIGS. 5B and 6C or anotherprocess is used to execute a process flow or processing job.

Turning to FIG. 6C, processing begins with process block 650, andproceeds to process block 652, wherein the current state is set to theSTART state. Next, in process block 654, the task associated with thecurrent state is attempted to be performed. As determined in processblock 656, if the task timed-out before completion, then as determinedin process block 658, if the task should be retried (e.g., the number ofretries specified in the process flow or a default value has not beenexhausted), processing returns to process block 656. Otherwise, inprocess block 660, the current state is updated to that corresponding tothe task's completion status (e.g., complete, non-complete,not-attempted, etc.). As determined in process block 662, if an erroroccurred (e.g., an invalid next state or other error condition), then anerror indication is returned to the selected request handler in processblock 664, and processing is complete as indicated by process block 666.Otherwise, if the next state is the FINISH state (as determined inprocess block 670), then the result and possibly a final set of stateinformation is sent to the selected request handler in process block672, and processing is complete as indicated by process block 672.Otherwise, in process block 674, the selected request handler is updatedwith current state information, such as, but not limited to (norrequired to include) the current state name, intermediate results,variable values, etc. Processing then returns to process block 654.

One embodiment of a hive uses a logical hierarchy of hive engines fordelegation of performing administrative and/or other hive related tasks.In one embodiment, each hive engine participates in the processingregion hierarchy as a region member with one hive engine in eachprocessing region being a region leader, and there one overall seniorregion leader for the hive. For example, shown in FIG. 7A are multipleprocessing regions 700-701, having an overall senior region leader 703(denoted senior leader/region leader/region member as it performs allfunctions) residing in processing region 700, a region leader/regionmember 707 in processing region 701, region members 704-705 inprocessing region 700, and region members 708-709 in processing region701.

FIGS. 7B-7C are flow diagrams of processes used in one embodiment toestablish and maintain this hierarchical relationship among distributedprocesses or systems, such as among hive engines. The generic terms ofheartbeat leader and heartbeat member are used in describing thisprocess, because it can be used in many different applications forestablishing and maintaining a hierarchical relationship in a set ofdynamic and autonomous processes and systems. For example, in oneembodiment, the processes illustrated in FIGS. 7B-C are used toestablish and maintain which hive engine in a region is the regionleader, and between region leaders for establishing which hive engine isthe senior region leader.

Processing of the heartbeat leader flow diagram illustrated in FIG. 7Bbegins with process block 720, and proceeds to process block 722 whereina multicast heartbeat request message is sent on the multicast addressbelonging to the group in which the hierarchical relationship is beingestablished and maintained. In process block 724, the responses arereceived. As determined in process block 725, if the process is seniorover those from which a response was received, then it remains theleader or senior process, and optionally in process block 726,piggybacked information (e.g., number of regions, number of members ineach region, etc.) is processed and possibly actions taken or initiatedin response. As indicated by process block 727, the process delays orwaits a certain period of time before repeating this process, and thenprocessing returns to process block 722. Otherwise, in process block728, the process demotes itself from being the leader or senior process(such as by initiating or switching to performing actions consistentwith being a region member if not already performing the functions of aregion member), and processing is complete as indicated by process block729.

Processing of the heartbeat member flow diagram illustrated in FIG. 7Cbegins with process block 740, and proceeds to process block 742,wherein the process watches for and identifies heartbeat requestmessages during a predetermined timeframe. As determined in processblock 744, if a no heartbeat request is received, then in process block745, the process promotes itself to being the heartbeat leader, andprocessing returns to process block 742. Otherwise, if this process issenior to a process sending a heartbeat request message as determined inprocess block 748, then processing proceeds to process block 745 topromotes itself. Otherwise, in process block 749, a heartbeat responsemessage is sent to the sender of the received heartbeat request message,and optionally other information is included in the heartbeat responsemessage. Processing then returns to process block 742. Note, determiningseniority can be performed in numerous manners and mechanisms, such asthat based on some physical or logical value associated with a hiveengine (e.g., one of its network addresses, its serial number, etc.)FIG. 8A illustrates some of the functions performed by a seniorprocessing region leader in one embodiment. Processing begins withprocess block 800, and proceeds to process block 802, wherein aheartbeat request is sent to all region leaders, typically by sending amulticast packet to the processing region leader intercommunicationmulticast address 242 (FIG. 2C) and piggybacked information is collectedfrom received responses with this information typically including, butnot limited to the number of processing regions, number of processinghandlers, number of request handlers, limited task information, etc. Asdetermined in process block 804, if the number of request handlers needsto be adjusted (e.g., there are too few or too many), then in processblock 806, a region leader is selected and directed to start or stop arequest handler. Next, as determined in process block 808, if the numberof processing regions needs to be adjusted (e.g., there are too few ortoo many), then in process block 810, a region leader is selected anddirected to disband or spit a region. Next, as determined in processblock 812, if the number of task handlers that can perform a particulartask (i.e., a “limited task” as typically and by default, all tasks canbe performed by all task handlers) needs to be adjusted (e.g., there aretoo few or too many), then in process block 814, a region leader isselected and directed to adjust the number of task handlers within itsregion which can perform the particular limited task. Next, asdetermined in process block 816, if some other action needs to beperformed, then in process block 818, the action is performed or aregion leader is instructed to perform the action. Next, processingusually waits or delays for a predetermined or dynamic amount of time asindicated by process block 819, before processing returns to processblock 802.

FIG. 8B illustrates some of the functions performed by a region leaderin one embodiment. Processing begins with process block 830, andproceeds to process block 832, wherein a heartbeat request is sent toall region member, typically by sending a multicast packet to theprocessing region multicast address 244 (FIG. 2C), and piggybackedinformation is collected from received responses with this informationtypically including, but not limited to the number of processinghandlers, number of request handlers, etc.; or possibly instructions arereceived from the senior region leader. As determined in process block834, if the number of request handlers needs to be adjusted (e.g., thereare too few or too many), then in process block 836, a process handleris selected and directed to start or stop a request handler. Next, asdetermined in process block 838, if the number of processing regionsneeds to be adjusted (e.g., there are too few or too many), then inprocess block 840, an instruction to disband or spit the region isissued. Next, as determined in process block 842, if the number of taskhandlers permitted to perform a particular limited task needs to beadjusted (e.g., there are too few or too many), then in process block844, an instruction is provided (directly, indirectly such as via arequest or process handler, or based on a volunteer pattern) to aparticular task handler to permit or deny it from performing theparticular limited task. Next, as determined in process block 846, ifsome other action needs to be performed, then in process block 848, theaction is performed or a process handler is instructed to perform theaction. Next, processing usually waits or delays for a predetermined ordynamic amount of time as indicated by process block 849, beforeprocessing returns to process block 832.

FIG. 8C illustrates the splitting of a region as performed in oneembodiment. Region leader 860 sends a multicast message 871 requesting avolunteer to head the new region to region members 861, some of whichtypically return a positive response message 872. Region leader 860 thenidentifies a selected region member 862 to head the new processingregion, and sends an appointment message 873 to selected region member862. In response, selected region member 862 creates a new processingregion as indicated by reference number 874, typically includingidentifying an unused processing region multicast address 244 (FIG. 2C)as it monitored the traffic or processing region active indicationmessages sent to processing region active region indications multicastaddress 243. Then, selected region member 862 multicasts a volunteermessage 875 to processing regions in the old (and still used) processingregion and typically receives one or more responses 876. Selected regionmember 862 then selects a certain number, typically half of the numberof process handlers in the old processing region, of responding processhandlers, and notifies them to switch to the new processing region viamove instruction 877, and they in turn, send a confirmation message 878to selected region member 862.

FIG. 9 illustrates a process used in one embodiment for initializing ahive engine. Processing begins with process block 900, and proceeds toprocess block 902, wherein a hive version request and hive joinmulticast message is sent typically to all region leaders. As determinedin process block 904, if no responses are received, then in processblock 912, a new processing region is formed, and request hander andregion leader processes are initiated. Next, in process block 914,process handler, recovery module, and region member processes areinitiated, and startup processing is completed as indicated by processblock 916. Otherwise, as determined in process block 906, if a hivesoftware update is available, then, in process block 908, one of theresponders is selected, the updates are acquired, and the software(e.g., hive software, operating system, etc.) is updated. In processblock 910, the hive engine joins the smallest or possibly one of thesmaller processing regions, possibly with this selection beingdetermined by identified territories, and processing proceeds to processblock 914.

In one embodiment, the hive is updated by a client with specialadministrative privileges. This administrative client sends a request tothe senior region leader of the hive. The senior region leader opens aTCP connection and sends the administration client the connectioninformation. The administration client sends the new application to thesenior region leader. When the senior region leader receives an update,the senior region leader multicasts the update command to all the hivemembers. The senior region leader sends multicast message containing thename of the file that is being updated, the new version, and the totalnumber of packets each hive member should receive. The senior regionleader then multicasts the data packets, each packet typically includesthe file id, the packet number, and data. If a hive member does notreceive a packet, that hive member sends a request to the senior regionleader for the missing packet. The senior region leader resends,multicasts, the missing packet. The hive members store the update in astaging area until they receive the activation command. To activate anupdate, the administration client sends the activation command to thesenior region leader. The senior region leader multicasts the activatecommand to the hive members. The hive members remove the old applicationor files and moves the update from the staging area to the productionarea. To update the hive software or operating system, the senior regionleader distributes the updates and restarts volunteers in a rollingfashion. When the hive service manager detects a new version of itself,the service manager forks the process and restarts with a new version.Also, the senior region leader can send other update commands. An activemessage indicates that the corresponding application, patch, or OS thatshould be running on the hive. A deactivated messages indicates that thecorresponding application, patch, or OS should not be running on thehive and should remain installed on hive members. A remove messageindicates that the corresponding application, patch, or OS was onceinstalled on the Hive and any instances found on Hive members should beremoved. This allows hive engines to be updated and also to move back toprevious releases.

In view of the many possible embodiments to which the principles of ourinvention may be applied, it will be appreciated that the embodimentsand aspects thereof described herein with respect to thedrawings/figures are only illustrative and should not be taken aslimiting the scope of the invention. For example and as would beapparent to one skilled in the art, many of the process block operationscan be re-ordered to be performed before, after, or substantiallyconcurrent with other operations. Also, many different forms of datastructures could be used in various embodiments. The invention asdescribed herein contemplates all such embodiments as may come withinthe scope of the following claims and equivalents thereof.

1. A system for processing information, the system comprising: a clientconfigured to cause the processing of a processing job in order todetermine a processing result; a plurality of request handlers; and aplurality of process handlers; wherein the client is configured toperform operations comprising: (a) sending to each of the plurality ofrequest handlers a client service request message for the processingjob; (b) receiving a plurality of responses to said sent client servicerequest message; (c) selecting a particular request handler of theplurality of request handlers based on said received responses to saidsent client service request message; and (d) receiving the processingresult from the particular request handler; wherein each of theplurality of request handlers is configured to perform operationscomprising: (a) sending a client service response message to a receivedsaid client service request message for the processing job; and (b) ifselected by the client to be the particular request handler to handlethe processing job: (1) querying the plurality of process handlers todetermine a particular process handler of the plurality of processhandlers to service the processing job based on a received plurality ofresponses from different process handlers, including the particularprocess handler, of the plurality of process handlers; (3) receiving theprocessing result from the particular process handler; and (4)forwarding said processing result to the client; and wherein each of theplurality of process handlers is configured to perform operationscomprising: (a) responding to said query from the particular requesthandler; and (b) if identified as the particular process handler by theparticular request handler: (1) performing the processing job todetermine the processing result; and (2) communicating the processingresult to the particular request handler.
 2. The system of claim 1,wherein the particular process handler is configured to performoperations including updating the particular request handler with stateinformation pertaining to partial processing of the processing job; andwherein the particular request handler is configured to performoperations including: receiving said state information pertaining to thepartial processing of the processing job; updating a data structure withsaid state information; and in response to a detected error condition,assigning the processing of the processing job to identify theprocessing result starting from a partial processing state identifiedfrom said state information to one of the plurality of process handlersdifferent from the particular process handler.
 3. The system of claim 1,where the processing job includes a reference or identification of aprocess flow, the process flow identifies a plurality of processingsets, each of the plurality of processing sets identifying a beginningstate, a task, and one or more ending states.
 4. The system of claim 3,wherein the particular process handler is configured to performoperations including updating the particular request handler with stateinformation pertaining to partial processing of the processing job atthe completion of each of one or more of the plurality of processingsets; and wherein the particular request handler is configured toperform operations including: receiving said state informationpertaining to the partial processing of the processing job; updating adata structure with said state information; and in response to adetected error condition, assigning the processing of the processing jobto identify the processing result starting from a partial processingstate identified from said state information to one of the plurality ofprocess handlers different from the particular process handler.
 5. Thesystem of claim 1, where the processing job includes a reference oridentification of a process flow; wherein the particular process handleris configured to perform operations including updating the particularrequest handler with state information pertaining to partial processingof the processing job at one or more points identified by the processflow; and wherein the particular request handler is configured toperform operations including: receiving said state informationpertaining to the partial processing of the processing job; updating adata structure with said state information; and in response to adetected error condition, assigning the processing of the processing jobto identify the processing result starting from a partial processingstate identified from said state information to one of the plurality ofprocess handlers different from the particular process handler.
 6. Thesystem of claim 1, wherein the particular process handler is configuredto perform operations including updating the particular request handlerwith state information pertaining to partial processing of theprocessing job at a plurality of different intermediate processingpoints; and wherein the particular request handler is configured toperform operations including: receiving said state informationpertaining to the partial processing of the processing job; updating adata structure with said state information; and in response to adetected error condition, assigning the processing of the processing jobto identify the processing result starting from a partial processingstate identified from said state information to one of the plurality ofprocess handlers different from the particular process handler.
 7. Thesystem of claim 1, including a plurality of computing devices, whereineach of the plurality of computing devices includes a different one ofsaid plurality of process handlers, and each of said plurality ofrequest handlers is included in a different one of the plurality ofcomputing devices.
 8. A networked system for processing information, thesystem comprising: a client computer including a client; and a pluralityof computers, each of the plurality of computers including a requesthandler, a process handler, or both a request handler and a processhandler; wherein each of said request handlers is configured to: send aresponse to a client service request received from the client if itdetermines it is available to handle a processing job identified in saidclient service request received from the client for the processing job,and if identified by the client as the selected request handler tohandle the processing job: to send one or more query messages to theplurality of process handlers, to receive responses from said processhandlers; to select among said received responses to identify and assigna particular process handler to service the processing job, to receive aprocessing result from the particular process handler, and to send aresult corresponding to the processing result to the client; whereineach of the process handlers is configured to: respond to said one ormore query messages if it is available to handle the processing job, andif identified as the particular process handler by the particularrequest handler: to process the processing job and to communicate theprocessing result to said identified request handler; and wherein theclient is configured to: receive a plurality of said responses to theclient service request; to select the selected request handler based onsaid received responses to the client service request; and to receivethe processing result from the selected request handler.
 9. The systemof claim 8, wherein the plurality of process handlers are differentprocesses than the plurality of request handlers.
 10. A system forprocessing information, the system comprising: a request regionincluding a plurality of request handlers, each of said request handlersbeing a separate process; and a plurality of processing regions, each ofthe plurality of processing regions including a plurality of processhandlers, each of the process handlers being a separate process; whereineach of the plurality of request handlers is configured to send aresponse if it determines it is available to handle a processing jobidentified in a client service request received from a client for theprocessing job, and if identified by the client to handle the processingjob: to send one or more query messages to one or more of the processingregions, to receive responses from said processing regions; and toselect among said received responses to identify and assign a particularprocess handler to service the processing job, and to receive aprocessing result from the particular process handler; wherein each ofthe process handlers within said one or more of the processing regionsis configured to respond to one of said one or more query messages if itis available to handle the processing job, and if identified as theparticular process handler by the particular process handler to processthe processing job to determine the processing result, and tocommunicate the processing result to said identified request handler;wherein the client is configured to receive a plurality of saidresponses from the client service request, and to receive the processingresult from the particular process handler; and wherein said operationof being identified by the client to handle the processing job includesthe client selecting the particular process handler based on saidreceived responses to the client service request.
 11. The system ofclaim 10, wherein the plurality of process handlers are differentprocesses than the plurality of request handlers.
 12. A system forprocessing information, the system comprising: a plurality of requesthandlers, each of said request handlers being a separate process; and aplurality of process handlers, each of the process handlers being aseparate process; wherein each of the plurality of request handlers isconfigured to send a response to a client service request received froma client if it determines it is available to handle a processing jobidentified in said client service request received from the client forthe processing job, and if identified by the client to handle theprocessing job: to send one or more query messages to the plurality ofprocess handlers, to receive responses from said process handlers; andto select among said received responses to identify and assign aparticular process handler to service the processing job, to receive aprocessing result from the particular process handler, and to send theprocessing result to the client; wherein each of the process handlers isconfigured to respond to said one or more query messages if it isavailable to handle the processing job, and if identified as theparticular process handler by the particular process handler: to processthe processing job and to communicate the processing result to saididentified request handler; and wherein the client is configured toreceive a plurality of said responses from the client service request,to select the particular process handler based on said receivedresponses from the client service request, and to receive the processingresult from said the particular process handler assigned to service theprocessing job.
 13. The system of claim 12, wherein the plurality ofprocess handlers are different processes than the plurality of requesthandlers.
 14. A method for processing information in a networked hive ofcomputing engines, the method comprising: each of a plurality of requesthandlers receiving a client service request message for a particular jobfrom a client; a plurality of the plurality of request handlersresponding to the client request message by sending a response messageto the client; a selected request handler of said request handlers saidresponding to client request message receiving instructions from theclient to process the particular job, wherein the client is external tothe computing engine on which the selected request handler resides; theselected request handler sending a processing request message for theparticular job to a plurality of process handlers; a plurality of theplurality of process handlers responding to the processing requestmessage by sending a response message to the selected request handler;the selected request handler selecting a selected process handler fromsaid process handlers said responding to the processing request messageand assigning the particular job to the selected process handler; theselected process handler, in response to said assignment by the selectedrequest handler, performing one or more tasks of the particular job togenerate a processing result, and sending the processing result to theselected request handler; and the selected request handler receiving theprocessing result and sending the processing result to the client;wherein the selected request handler and the selected process handlerare on different computing engines.
 15. A method for processinginformation in a networked hive of computing engines, the methodcomprising: each of a plurality of request handlers receiving a clientservice request message for a particular job from a client; a pluralityof the plurality of request handlers responding to the client requestmessage by sending a response message to the client; a selected requesthandler of said request handlers said responding to client requestmessage receiving instructions from the client to process the particularjob, wherein the client is external to the computing engine on which theselected request handler resides; the selected request handler sending aprocessing request message for the particular job to a plurality ofprocess handlers; a plurality of the plurality of process handlersresponding to the processing request message by sending a responsemessage to the selected request handler; the selected request handlerselecting a selected process handler from said process handlers saidresponding to the processing request message and assigning theparticular job to the selected process handler; the selected processhandler, in response to said assignment by the selected request handler,performing partial processing of the particular job to generate stateinformation, and sending the state information to the selected requesthandler; the selected request handler, subsequent to receiving saidstate information, assigning the particular job to a second processhandler of the plurality of process handlers and communicating saidstate information to the second process handler; the second processhandler completing processing of the particular job from its partiallyprocessed state identified from said state information to generate aprocessing result, and sending the processing result to the selectedrequest handler; and the selected request handler receiving theprocessing result and sending the processing result to the client;wherein the selected request handler, the selected process handler, andthe second process handler are on different computing engines.
 16. Themethod of claim 15, wherein said operation by the selected requesthandler of assigning the particular job to the second process handler isperformed in response to the selected request handler detecting anerror.
 17. The method of claim 15, comprising: the selected requesthandler performing a volunteer operation to select the second processhandler based on received responses from a plurality of the plurality ofprocess handlers including the second process handler.
 18. The method ofclaim 17, wherein said operation by the selected request handler ofassigning the particular job to the second process handler is performedin response to the selected request handler detecting an error.
 19. Thesystem of claim 18, where the particular job includes a reference oridentification of a process flow, the process flow identifies aplurality of processing sets, each of the plurality of processing setsidentifying a beginning state, a task, and one or more ending states;wherein said state information identifies the state of the particularjob at one of said ending states.
 20. The system of claim 15, where theparticular job includes a reference or identification of a process flow,the process flow identifies a plurality of processing sets, each of theplurality of processing sets identifying a beginning state, a task, andone or more ending states; wherein said state information identifies thestate of the particular job at one of said ending states.
 21. A methodfor processing information in a networked hive of computing engines, themethod comprising: a request handler of a plurality of request handlersreceiving instructions from a client to process the particular job,wherein the client is external to the computing engine on which therequest handler resides; the request handler sending a processingrequest message for the particular job to a plurality of processhandlers; a plurality of the plurality of process handlers responding tothe processing request message by sending a response message to therequest handler; the request handler selecting a selected processhandler from said process handlers said responding to the processingrequest message and assigning the particular job to the selected processhandler; the selected process handler, in response to said assignment bythe request handler, performing one or more tasks of the particular jobto generate a processing result, and sending the processing result tothe request handler; and the request handler receiving the processingresult and sending the processing result to the client; wherein therequest handler and the selected process handler are on differentcomputing engines.